Multisensory integration with a head-mounted display and auditory display

Human operators who use head-mounted displays (HMDs) in their work may benefit from auditory support. It is unclear whether auditory support is better delivered in free-field or via earpiece, and what the effect of walking is. To examine this problem, a novel multisensory integration task was created in which participants identified mismatches between sounds and visual information on an HMD. Participants listened to the sounds either via earpiece or free-field while they either sat or walked about the test room. When using an earpiece, participants performed the mismatch task equally well whether walking or sitting, but when using free-field sound, participants performed the task significantly worse when walking than when sitting. The worse performance for participants using free-field sound while walking may relate to spatial and motion inconsistencies between the sound and vision or because of misperceptions of the time at which the sounds occurred. The results underscore the need for representative design of experiments exploring multisensory integration and they suggest auditory conditions that might influence effective multisensory integration with HMDs.